Skip to content

feat(voice): add AssemblyAI streaming STT provider#1912

Open
cjol wants to merge 55 commits into
mainfrom
fix/jul9-1900-assemblyai
Open

feat(voice): add AssemblyAI streaming STT provider#1912
cjol wants to merge 55 commits into
mainfrom
fix/jul9-1900-assemblyai

Conversation

@cjol

@cjol cjol commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

This PR adds @cloudflare/voice-assemblyai, a streaming speech-to-text provider for @cloudflare/voice, based on the contributor branch linked in #1900. Closes #1900.

Why

  • @cloudflare/voice already supports pluggable STT providers, but users who want AssemblyAI streaming transcription need to write their own adapter.
  • The provider needs to handle AssemblyAI realtime-specific behavior, including WebSocket upgrade, turn events, speech-start events, frame sizing, and optional agent context carryover.
  • The framework-level change keeps context carryover optional on TranscriberSession rather than coupling it to a specific provider. Existing providers can ignore it.

Public API Surface

Symbol Kind Notes
AssemblyAISTT class New STT provider exported by @cloudflare/voice-assemblyai.
AssemblyAISTTOptions type Provider options for API key, model, latency mode, prompt, keyterms, language settings, noise suppression, endpoint, and turn behavior.
TranscriberSession.updateAgentContext optional method Allows providers to receive spoken agent replies after TTS output.

Code Changes

  • Adds voice-providers/assemblyai, following the existing voice provider package structure.
  • Maps AssemblyAI realtime turn events into onInterim, onUtterance, and onSpeechStart callbacks.
  • Coalesces short audio frames so telephony integrations meet AssemblyAI message-size requirements.
  • Adds optional agent-context updates after agent speech in withVoice.
  • Updates Workers AI TTS error handling so non-OK responses are surfaced as errors instead of forwarded as audio bytes.
  • Adds an examples/assemblyai-voice-agent reservation-agent demo.
  • Documents the new provider in the voice package docs and README.

Open in Devin Review

@changeset-bot

changeset-bot Bot commented Jul 9, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: f169317

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@cloudflare/voice Patch
@cloudflare/voice-assemblyai Patch
@cloudflare/voice-elevenlabs Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

devin-ai-integration[bot]

This comment was marked as resolved.

Comment thread voice-providers/assemblyai/package.json Outdated
@cjol cjol marked this pull request as draft July 13, 2026 08:30
Spec for @cloudflare/voice-assemblyai — a type-compliant Streaming v3
STT provider for the Agents voice pipeline, mirroring the Deepgram/Telnyx
example. Captures scope (STT-only, baseUrl-configurable for AI Gateway),
the v3 protocol mapping, test plan, and deferred work (typed gateway
helper, Workers AI self-host, on-prem pharma).
Apply design-review simplifications:
- options down to apiKey/model/formatTurns/medical/keyterms/baseUrl/params
- add `params` passthrough; drop endOfTurnConfidenceThreshold
- hardcode encoding=pcm_s16le and sample_rate=16000 (pipeline-fixed)
- drop `region` (use baseUrl for EU/gateway); document URLs in README
- cut `prompt` from v1; close() no longer awaits Termination
- single test file; add Known Limitations (no onError, model-determined language)
…ence

Verified streaming params against AssemblyAI's live API docs (via MCP):
- lock model to u3-rt-pro (the voice-agent model); drop the `model` option
- auth via Authorization header (raw key, no prefix), NOT a ?token= query param
- drop format_turns (u3-rt-pro finals are always formatted)
- fully typed surface, no passthrough: domain, keyterms, minTurnSilence,
  maxTurnSilence, interruptionDelay, continuousPartials, baseUrl
- rename medical:boolean -> domain:"medical-v1"|(string&{})
- hardcode speech_model/sample_rate/encoding; conditional params only when set
- limitations: no onError, language model-determined, single model / 6 languages
Verified against AssemblyAI prompting + streaming API docs:
- prompt is a connection param (no UpdateConfiguration round-trip needed);
  it is the u3-rt-pro differentiator and the language-guidance lever. Doc the
  "omit for the optimized default prompt" recommendation.
- vadThreshold (vad_threshold): VAD silence-confidence threshold for noisy envs.
- both appended only when set; language limitation softened (guided via prompt).
language_detection applies to u3-rt-pro (native multilingual code-switching) and
returns language_code/language_confidence on Turn events. Since the Cloudflare
transcript callbacks are text-only, pair the flag with a provider-specific
onLanguageDetected(languageCode, languageConfidence) callback to surface it
without changing the shared interface.
- Add a `speechModel` option (defaults to u3-rt-pro) with the `AssemblyAISpeechModel` type and model-aware validation that rejects u3-rt-pro-only params (prompt, continuousPartials, interruptionDelay) on universal-streaming models.
- Default min/max turn silence to 400/1280 ms (above the server's 100/1000) so speakers can pause mid-thought without the turn being cut off; callers can still override.
- Default continuous_partials on for u3-rt-pro for a live mid-turn transcript.
- Log unexpected WebSocket closes (auth/session errors only arrive as close frames) and rewrite wss:// to https:// for Cloudflare's fetch-based upgrade.
- Streamline the intro, usage, options table, and how-it-works sections; document the new `speechModel` option and the 400/1280 ms turn-silence defaults.
- Add an "AssemblyAI documentation" section linking the relevant streaming pages (Universal-3 Pro, turn detection & partials, message sequence, streaming API, endpoints & data zones, Medical Mode, keyterms) and a dashboard link for the API key.
Real-time voice agent running entirely in a Durable Object: AssemblyAI u3-rt-pro streaming STT, Workers AI LLM (glm-4.7-flash) with tools, and Workers AI MeloTTS for keyless TTS. Browser mic streams 16 kHz PCM over a plain WebSocket via the useVoiceAgent React hook, with barge-in, streaming responses, and conversation memory. Only ASSEMBLYAI_API_KEY is required.
Update the AssemblyAI streaming provider to the current model and API.

Plugin (voice-providers/assemblyai):
- Lock speech_model to universal-3-5-pro; drop the speechModel option and
  the old model-compat assertions (universal-3-5-pro is is_u3_pro server-side,
  so it supports prompt/agent_context/mode/etc).
- Add connection params: mode, agentContext, previousContextNTurns,
  languageCode, voiceFocus, voiceFocusThreshold, validated against the
  server's ranges (1500-char caps, threshold requires voiceFocus,
  interruptionDelay 0-1000, previousContextNTurns 0-100).
- Defer turn-silence / partials to mode instead of force-defaulting them.
- Add session.updateAgentContext(text): sends an UpdateConfiguration with
  agent_context mid-session (pre-connect buffering, latest-only, 1500 cap).

Framework (@cloudflare/voice):
- Add optional TranscriberSession.updateAgentContext?(text) (no-op for
  providers without context carryover), a forwarder on
  AudioConnectionManager, and pipeline calls so withVoice feeds each spoken
  reply/greeting back to the transcriber as conversational context.

Also update the example, provider/voice READMEs, and voice docs.
When previousContextNTurns is 0, the server discards agent_context (the
carryover window is zero), so AssemblyAISession.updateAgentContext() now
early-returns instead of sending an UpdateConfiguration the server would
throw away. Contained to the plugin — the framework seam still calls
updateAgentContext() unconditionally and the provider decides to no-op.
Rename prose references from "Universal-3.5 Pro Streaming" to the official
"Universal 3.5 Pro Realtime" across the provider, example, READMEs, and voice
docs. The wire value (speech_model=universal-3-5-pro) is unchanged.
- prompt and keyterms are NOT mutually exclusive on streaming u3.5-pro (no
  server validator enforces it) — correct the docs and the prompt JSDoc.
- Raise MAX_PROMPT_CHARS 1500 -> 1750 to match the current API
  (connection_parameters.py); applies to prompt, agentContext, and the
  updateAgentContext() truncation.
- Add descriptors for the Prompting/Keyterms, Conversation Context, and
  Voice Focus documentation links.
The "Endpoints & data zones" link pointed at the streaming API parameter
reference; repoint it to /docs/streaming/endpoints-and-data-zones, which
actually lists the regional WebSocket hosts the descriptor describes.
AssemblyAI requires 50-1000ms of audio per binary WebSocket message and
terminates the session (3007) on the second undersized message. Browser
mic chunks are 100ms and pass through untouched, but the Twilio/Telnyx
adapters relay 20ms frames, which killed the session almost immediately.
Accumulate sub-minimum frames until they cross 1600 bytes (50ms at 16kHz
mono s16le), matching AssemblyAI's own LiveKit adapter. A sub-minimum
tail is dropped at close().
dlange-aai and others added 18 commits July 13, 2026 11:05
Say 'server default' for previousContextNTurns instead of a number the
docs don't pin down, source the keep-context-concise guidance from the
Conversation Context page, and cite session-based billing rather than
asserting the mechanics.
…g LLM errors

onCallStart queried cf_voice_messages with raw SQL, which throws 'no such
table' on a brand-new agent instance (the voice mixin creates the table
lazily) and silently killed the greeting. Use getConversationHistory(),
which ensures the schema. Also add a streamText onError logger — the AI
SDK otherwise swallows LLM failures, surfacing them only as an empty
reply ('No response generated').
The voice pipeline warns that textStream can join non-adjacent text
parts incorrectly (visible as doubled tokens); the other voice examples
already return result.fullStream.
glm-4.7-flash was intermittently hanging ~60s and returning 3043/504
from the inference upstream, leaving the agent silent. gpt-oss-20b (one
of the voice-agent example's endorsed models) streams in ~1s and handles
the tool-calling demo reliably.
…adapter

MeloTTS was intermittently failing with 3043 inference errors, leaving
the agent silent after transcription succeeded. The framework's default
WorkersAITTS (@cf/deepgram/aura-1) is the same path the other voice
examples use, and dropping the custom adapter simplifies the example.
…reply

gpt-oss-20b emits reasoning tokens before text; with Workers AI's ~256
default cap, longer reasoning consumed the whole budget and the turn
ended with finishReason=length and zero speakable text — surfacing as a
silent 'No response generated'. Cap at 2048 and log turn transcripts and
LLM finish stats so this failure mode is visible in the terminal.
Tested against the live stack: gpt-oss-20b works but its reasoning phase
delays the first spoken token and can starve short caps; glm-4.7-flash
was intermittently erroring upstream; llama-3.3-70b leaks tool calls as
literal JSON text through workers-ai-provider. llama-4-scout executes
tools correctly, emits no reasoning phase, and keeps replies concise.
…demo

Replace the generic time/weather assistant with Luna Rossa, a restaurant
reservation agent — a real use-case shaped around what the integration
demonstrates: terse answers after agent questions (agent_context
carryover), venue vocabulary (prompt + keyterms), reservations persisted
in the DO's SQLite across calls, and availability/booking/lookup/cancel
tools.

The LLM is OpenAI gpt-4.1-mini via the AI SDK: every Workers AI model
tested had a blocking defect in this multi-tool voice loop (glm-4.7-flash
upstream errors, llama tool-call JSON leaks and hallucinated confirmation
codes, gpt-oss reasoning-only empty replies, mistral token doubling).
Verified end-to-end: 4-turn spoken booking conversation with real tool
execution and a database-backed confirmation code.
Add an 'Under the hood' panel to the Luna Rossa client that renders
debug_event messages broadcast by the agent: the exact agent_context
values sent to AssemblyAI (observed by wrapping the transcriber's
updateAgentContext), and every tool call and result from onStepFinish.
Makes the integration's invisible machinery — context carryover and
database-backed tools — visible during a live call.
Replace WorkersAITTS (aura-1) with a small in-example CartesiaTTS adapter
— a natural, low-latency voice and a demonstration of bringing your own
TTS vendor to the pipeline via the TTSProvider interface. Requests are
serialized through a queue with one retry on 429, since Cartesia plans
cap concurrent requests and the pipeline synthesizes sentences in
parallel (an unqueued burst silently dropped a sentence mid-reply).

With the LLM on OpenAI and TTS on Cartesia, nothing uses the AI binding
anymore — drop it from wrangler.jsonc, so local dev needs no Cloudflare
login, just the three vendor keys.
…tions

Per-sentence synthesis over the REST endpoint made each sentence an
independent generation, so the voice's pitch and pacing jumped audibly
at sentence seams — Cartesia documents this exact failure mode and its
fix: stream sentences of one reply into a shared WebSocket context
(continuations), so the model extends one generation instead of
restarting.

The pipeline starts sentence syntheses eagerly, so the reply's first
sentence opens the context and yields all of its audio while later
sentences join it and yield nothing; a short idle timer finalizes the
context and barge-in cancels it. WS audio is raw PCM, wrapped per-chunk
in WAV headers for the client's decodeAudioData path (which also decodes
exactly, avoiding MP3 padding at seams). One-shot utterances (the
greeting) keep the simpler REST/MP3 path.
…pter

Keep the example focused on the AssemblyAI integration: the built-in
Workers AI TTS (aura-1) needs no extra vendor key or adapter code. The
Cartesia WebSocket-continuations adapter lives in branch history
(bb9cd02) as a reference for a future @cloudflare/voice-cartesia
provider. MeloTTS was evaluated as the non-Deepgram alternative but
currently fails ~half of requests upstream (3043).
Two robustness gaps from the final review: (1) if the socket never
connects (e.g. a bad API key), feed() buffered mic audio unboundedly in
the Durable Object — now capped at ~30s with a single logged warning;
(2) WebSocket.send() throws in the gap between the server closing the
socket and the close event firing — sends now route through a guarded
helper instead of letting the exception escape into the audio pipeline.
… audio

synthesize() read response.arrayBuffer() without checking response.ok,
so an error body (e.g. the Workers AI free-tier 429 quota JSON) was sent
to the client as audio bytes and failed to decode silently. Log the
status and body and return null so the pipeline skips the sentence.
Identify traffic from this provider with AssemblyAI's structured
user-agent format (AssemblyAI/1.0 (integration=Cloudflare-Agents)),
matching how the official SDKs and other integrations attribute usage.
@cjol cjol force-pushed the fix/jul9-1900-assemblyai branch from 0281b2d to 8ec361c Compare July 13, 2026 10:24
@pkg-pr-new

pkg-pr-new Bot commented Jul 13, 2026

Copy link
Copy Markdown

Open in StackBlitz

agents

npm i https://pkg.pr.new/agents@1912

@cloudflare/ai-chat

npm i https://pkg.pr.new/@cloudflare/ai-chat@1912

@cloudflare/codemode

npm i https://pkg.pr.new/@cloudflare/codemode@1912

create-think

npm i https://pkg.pr.new/create-think@1912

hono-agents

npm i https://pkg.pr.new/hono-agents@1912

@cloudflare/shell

npm i https://pkg.pr.new/@cloudflare/shell@1912

@cloudflare/think

npm i https://pkg.pr.new/@cloudflare/think@1912

@cloudflare/voice

npm i https://pkg.pr.new/@cloudflare/voice@1912

@cloudflare/worker-bundler

npm i https://pkg.pr.new/@cloudflare/worker-bundler@1912

commit: 976feed

@cjol

cjol commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@dlange-aai I've opened a PR from your branch for the AssemblyAI adapter, and I'm ready to merge it. I'm afraid I've pulled out the example, though, in an effort to start to standardise and simplify our various voice examples. The consolidated example is much more generic (doesn't have a particular scenario like the reservation desk), but lets the user set keywords, context, etc. Let me know if there's some feature that you think we're not showing off enough!

@cjol cjol marked this pull request as ready for review July 13, 2026 16:02

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

View 1 additional finding in Devin Review.

Open in Devin Review

Comment thread voice-providers/elevenlabs/src/index.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proposal: AssemblyAI streaming STT voice provider (@cloudflare/voice-assemblyai)

3 participants